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The  Scene 

You’re  in  an  alternate  universe... 

You  may  be  sick... 

No  hospitals.. .no  general  practitioners.. .no  AMA... 

“Doctors”  are  self-declared... 

But  there  are  lots  of  people  who  have  designed  and 
built  blood  labs...MRIs...EKGs... 

And  they  all  want  you  to  use 
their  diagnostic  tool! 
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You’re  Living  There  Now... 

There  are  no  “general  practitioners”  to  go  to... 

There  is  no  “AMA”  of  consultants  to  acquisition 
programs... 

There  are  many  choices  of  risk-based  diagnostics: 
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We  have  to  do  better  than  this! 

We  need  general  practitioners  to 

•  help  acquisition  programs  understand  when  their 
symptoms  are  not  “normal” 

•  recommend  appropriate  diagnostics 

•  guide  programs  to  appropriate  interventions  or 
toward  “healthy  lifestyles” 

These  GPs  need 

•  knowledge  of  various  diagnostics — pros  &  cons 

•  guidance  in  choosing  any  particular  diagnostic  or 
sequence  of  diagnostics 

•  a  “patient  file,”  kept  over  time,  that  includes  the 
diagnostic  results 
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The  SEI  Chief  Engineer 

The  SEI  has  four:  Army,  Navy,  Air  Force,  Civil/Intel 
They  each  have 

*  Education,  experience,  knowledge  and  expertise  in  a 
broad  range  of  technologies,  disciplines,  areas 

*  In-depth  knowledge  in  particular  areas,  but  not  in  all  areas 

*  Interest  in  the  overall  health  of  all  programs  in  their 
“practice”  — unbiased,  detached,  impartial 

They  are  ideally  placed  to  become  the  “general  practitioners” 
we  need — they  just  need  more  complete  “reference 
materials” 

Other  organizations  can  take  on  a  similar  role  as  well,  but 
everyone  needs  a  “roadmap” 
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Diagnostics — Two  Kinds 


Model  Based: 

Risk  Based: 

Focused  on  how 

Focused  on  what  you 

things  should  be  and 

are  doing  and  what 

how  much  you  deviate 

risks  you  run  in 

from  that 

continuing — should 

model — should  create 

create  risk  items 

findings 


We  have  chosen  to  only  look  at  risk-based 
diagnostics  in  this  initial  roadmap  work. 

Model-based  diagnostic  can  be  added  later. 
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Context  for  Risk  Items — CMMI® 
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Primary  Sources  of  Risk  Items 
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How  did  we  identify  the  initial 
diagnostic  methods  that  should  be 
included  in  the  roadmap? 

We  started  with  the  ones  we  knew. 

If  we  could  explain  what  characteristics  qualified  them 
to  be  on  the  list,  we  could  then  go  out  and  find  others 
like  them. 

These  emerged  as  the  key  qualifiers: 

•  Risk  identification  phase 

•  Analysis  phase 

•  Potential  risk  statement  “leave  behind” 
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Risk-Based  Diagnostics 


—  —  ■  Risk-based  Diagnostic  —  —  —  —  -| 


Per  CMMI 
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First  Three  Diagnostic  Methods 

SRE:  Software  Risk  Evaluation — facilitation-based 

process  to  document  and  analyze  all  risks  already 
known  by  people  in  a  project 

ATAM:  Architecture  Tradeoff  Analysis  Method — scenario- 
driven  process  to  analyze  a  proposed  or  existing 
system  architecture  for  inherent  risks 

CURE:  COTS  Usage  Risk  Evaluation — rules-driven 
process  to  identify  and  analyze  the  risks  in  a 
project’s  application  of  COTS  products 
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Creating  the  Roadmap 

•  Follow  the  medical  analogy 

•  Begin  the  “roadmap”  for  risk-based 
diagnostics  with  at  least  3  entry  points: 

-  “Thermometer”  (self  diagnosis) 

-  “Routine  Physical” 

-  “Emergency” 

•  Define  a  reasonable  number  of  exit  points  (5  so  far) 

•  Plug  in  the  diagnostics  we  know  and  understand  today  in 
some  reasonable  sequence 

•  Find  other  risk-based  diagnostics  that  meet  our  definition 
and  plug  them  in 

•  Identify  missing  discriminators  and  currently  undefined 
diagnostics 
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Notional  Roadmap 


Entry  Point 

1 : 

Self 

Diagnosis 

Entry  Point 
2: 

Routine 

Physical 


Entry  Point 

3: 

Emergency 
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Lifestyle 

Changes 

Needed 


Come  Back 
Next  Year 


Extreme 

Measures 

Needed 


A 


Out  of 
Immediate 
Danger 


Terminal 

Condition 
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Entry  Points 
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Entry  Point  1— ’'Thermometer” 


m 


&  h 


Program  recognizes  symptoms  of  potential 
problems  H\ 

;  \  ;  /■ 

Program  administers  self-diagnostic  tool — no 
expert  needed  ,  \ , .  •• ^  '\ 

/V,  1  /  :'T"'  /  \ 

Tool  provides-a  reading 

•  Easy  to  interpret  ' 

•  Single  “snapshot”  easurement 

•  Based  on  generally  accepted  picture  of  health 


V3 


V-  V  -  ::  ‘ 


May  seek  help  depending  on  reading  (temperature) 

If  results  are  sufficiently  alarming,  will  go  to  their 
own  GP  or  seek  one  out 
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Entry  Point  2 — Checkup/Physical 

Program  (“patient”)  not  necessarily  feeling  “sick” 

Program  enlists  the  assistance  of  GP  to  maintain 
good  program  health;  may  or  may  not  have  a  prior 
relationship  with  the  GP 

GP  uses  interview  and  simple  risk-based 
diagnostics  (TBD — may  be  surveys,  checklists), 
and  may  prescribe  more  costly  risk-based 
diagnostics  "-■■■■ 

GP  reviews  diagnostic  results  and  decides 
whether  to  recommend  aggressive  “treatment” 

GP  may  opt  for  emergency  treatment  at  any  time 
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Entry  Point  3 — Emergency 

Program  (“patient’’)  knows  something  is  wrong 
and  requires  immediate  attention  (critical  care) 

Program  calls  in  the  <5P  for  intervention 
(“emergency  rooms”  don’t  exist  yet) 

_ ^  \  II 

GP  recommends  course  of  action  to  address 


immediate  problem 
“Patient  record§*ta 


Further  assessment  needed  to  determine  long¬ 
term  health  plan 
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Exit  Points 


Come  Back  Next  Year 

No  further  diagnostics  needed;  keep  on  doing  what 
you’re  doing 

Lifestyle  Changes  Needed 

No  immediate  danger,  but  you’re  headed  for  trouble 

Extreme  Measures  Needed 

You  can  be  saved,  but  we  have  to  act  fast 

Out  of  Immediate  Danger 

We’ve  saved  you  for  now;  go  back  to  your  GP’s  care 

Terminal  Condition 

Recovery  is  not  possible;  cut  losses  and  terminate 
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The  Risk  Item 

So  far,  only  the  SRE  defines  the  structure  of  the  risk  item,  so  it 
becomes  our  interim  model 


The  SRE  risk  item  (“Risk  Statement”): 

•  a  factual  condition  statement,  followed  by 

•  at  least  one  possible  consequence  of  that  condition 

•  supplemented  with  context  for  complete  understanding. 


|_Context _ 

“There  is  water  on  the  hall  floor;  someone  could  slip  and  fall.” 
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What’s  next 

Collaborate  with  others  to  identify  additional  methods  to  be 
included  in  the  Roadmap;  first  candidates: 

-  Software  Quality  Assessment  Exercise  (SQAE  - 
developed  by  MITRE) 

-  Operationally  Critical  Threat,  Asset,  and 
Vulnerability  Evaluation  (OCTAVE®) 

Cast  our  net  wider:  solicit  risk  based  diagnostics  from  all 
other  sources 

Put  Roadmap  in  the  hands  of  Chief  Engineers  and  other 
agencies  consulting  to  government  acquisition  programs  for 
validation 
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Schedule  and  Deliverables 


10/1/2003 

Team  Charter  and  Approach  Defined 


1/27/2004 

Diagnostic  Roadmap  Presentation 


i 


Characterization  of  Diagnostic  Techniques 
High  Level  Desert  " 


11/1/2003  12/1/2003 


1/1/2004 


6/1/2004 

4/1/2004  Publish  Diagnostic  Roadmap  V 1 .0 
Draft  Version  of  Diagnostic  Roadmap 


1727/2004  -  3/31/2004 
Review  of  add’l  candidate 
diagnostics  for  inclusion 


4/1/2004-5/31/2004 
Revise  Draft  Roadmap 


2/1/2004  3/1/2004 


4/1/2004  5/1/2004 


6/1/2004 


10/1/2003 
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